FACET: Fairness in Computer Vision Evaluation Benchmark

Computer vision models have known performance disparities across attributes such as gender and skin tone. This means during tasks such as classification and detection, model performance differs for certain classes based on the demographics of the people in the image. These disparities have been shown to exist, but until now there has not been a unified approach to measure these differences for common use-cases of computer vision models. We present a new benchmark named FACET (FAirness in Computer Vision EvaluaTion), a large, publicly available evaluation set of 32k images for some of the most common vision tasks - image classification, object detection and segmentation. For every image in FACET, we hired expert reviewers to manually annotate person-related attributes such as perceived skin tone and hair type, manually draw bounding boxes and label fine-grained person-related classes such as disk jockey or guitarist. In addition, we use FACET to benchmark state-of-the-art vision models and present a deeper understanding of potential performance disparities and challenges across sensitive demographic attributes. With the exhaustive annotations collected, we probe models using single demographics attributes as well as multiple attributes using an intersectional approach (e.g. hair color and perceived skin tone). Our results show that classification, detection, segmentation, and visual grounding models exhibit performance disparities across demographic attributes and intersections of attributes. These harms suggest that not all people represented in datasets receive fair and equitable treatment in these vision tasks. We hope current and future results using our benchmark will contribute to fairer, more robust vision models. FACET is available publicly at https://facet.metademolab.com

Zig-Zag Numberlink is NP-Complete

When can $t$ terminal pairs in an $m \times n$ grid be connected by $t$ vertex-disjoint paths that cover all vertices of the grid? We prove that this problem is NP-complete. Our hardness result can be compared to two previous NP-hardness proofs: Lynch's 1975 proof without the ``cover all vertices'' constraint, and Kotsuma and Takenaga's 2010 proof when the paths are restricted to have the fewest possible corners within their homotopy class. The latter restriction is a common form of the famous Nikoli puzzle \emph{Numberlink}; our problem is another common form of Numberlink, sometimes called \emph{Zig-Zag Numberlink} and popularized by the smartphone app \emph{Flow Free}

Beyond web-scraping: Crowd-sourcing a geographically diverse image dataset

Current dataset collection methods typically scrape large amounts of data from the web. While this technique is extremely scalable, data collected in this way tends to reinforce stereotypical biases, can contain personally identifiable information, and typically originates from Europe and North America. In this work, we rethink the dataset collection paradigm and introduce GeoDE, a geographically diverse dataset with 61,940 images from 40 classes and 6 world regions, and no personally identifiable information, collected through crowd-sourcing. We analyse GeoDE to understand differences in images collected in this manner compared to web-scraping. Despite the smaller size of this dataset, we demonstrate its use as both an evaluation and training dataset, highlight shortcomings in current models, as well as show improved performances when even small amounts of GeoDE (1000 - 2000 images per region) are added to a training dataset. We release the full dataset and code at https://geodiverse-data-collection.cs.princeton.edu

The Theory of the Interleaving Distance on Multidimensional Persistence Modules

In 2009, Chazal et al. introduced $\epsilon$-interleavings of persistence modules. $\epsilon$-interleavings induce a pseudometric $d_I$ on (isomorphism classes of) persistence modules, the interleaving distance. The definitions of $\epsilon$-interleavings and $d_I$ generalize readily to multidimensional persistence modules. In this paper, we develop the theory of multidimensional interleavings, with a view towards applications to topological data analysis. We present four main results. First, we show that on 1-D persistence modules, $d_I$ is equal to the bottleneck distance $d_B$. This result, which first appeared in an earlier preprint of this paper, has since appeared in several other places, and is now known as the isometry theorem. Second, we present a characterization of the $\epsilon$-interleaving relation on multidimensional persistence modules. This expresses transparently the sense in which two $\epsilon$-interleaved modules are algebraically similar. Third, using this characterization, we show that when we define our persistence modules over a prime field, $d_I$ satisfies a universality property. This universality result is the central result of the paper. It says that $d_I$ satisfies a stability property generalizing one which $d_B$ is known to satisfy, and that in addition, if $d$ is any other pseudometric on multidimensional persistence modules satisfying the same stability property, then $d\leq d_I$. We also show that a variant of this universality result holds for $d_B$, over arbitrary fields. Finally, we show that $d_I$ restricts to a metric on isomorphism classes of finitely presented multidimensional persistence modules.Comment: Major revision; exposition improved throughout. To appear in Foundations of Computational Mathematics. 36 page

The effectiveness of MAE pre-pretraining for billion-scale pretraining

This paper revisits the standard pretrain-then-finetune paradigm used in computer vision for visual recognition tasks. Typically, state-of-the-art foundation models are pretrained using large scale (weakly) supervised datasets with billions of images. We introduce an additional pre-pretraining stage that is simple and uses the self-supervised MAE technique to initialize the model. While MAE has only been shown to scale with the size of models, we find that it scales with the size of the training dataset as well. Thus, our MAE-based pre-pretraining scales with both model and data size making it applicable for training foundation models. Pre-pretraining consistently improves both the model convergence and the downstream transfer performance across a range of model scales (millions to billions of parameters), and dataset sizes (millions to billions of images). We measure the effectiveness of pre-pretraining on 10 different visual recognition tasks spanning image classification, video recognition, object detection, low-shot classification and zero-shot recognition. Our largest model achieves new state-of-the-art results on iNaturalist-18 (91.3%), 1-shot ImageNet-1k (62.1%), and zero-shot transfer on Food-101 (96.2%). Our study reveals that model initialization plays a significant role, even for web-scale pretraining with billions of images

A fractal dimension for measures via persistent homology

We use persistent homology in order to define a family of fractal dimensions, denoted $\mathrm{dim}_{\mathrm{PH}}^i(\mu)$ for each homological dimension $i\ge 0$, assigned to a probability measure $\mu$ on a metric space. The case of $0$-dimensional homology ($i=0$) relates to work by Michael J Steele (1988) studying the total length of a minimal spanning tree on a random sampling of points. Indeed, if $\mu$ is supported on a compact subset of Euclidean space $\mathbb{R}^m$ for $m\ge2$, then Steele's work implies that $\mathrm{dim}_{\mathrm{PH}}^0(\mu)=m$ if the absolutely continuous part of $\mu$ has positive mass, and otherwise $\mathrm{dim}_{\mathrm{PH}}^0(\mu)<m$. Experiments suggest that similar results may be true for higher-dimensional homology $0<i<m$, though this is an open question. Our fractal dimension is defined by considering a limit, as the number of points $n$ goes to infinity, of the total sum of the $i$-dimensional persistent homology interval lengths for $n$ random points selected from $\mu$ in an i.i.d. fashion. To some measures $\mu,$ we are able to assign a finer invariant, a curve measuring the limiting distribution of persistent homology interval lengths as the number of points goes to infinity. We prove this limiting curve exists in the case of $0$-dimensional homology when $\mu$ is the uniform distribution over the unit interval, and conjecture that it exists when $\mu$ is the rescaled probability measure for a compact set in Euclidean space with positive Lebesgue measure

Convalescent plasma in patients admitted to hospital with COVID-19 (RECOVERY): a randomised controlled, open-label, platform trial

SummaryBackground Azithromycin has been proposed as a treatment for COVID-19 on the basis of its immunomodulatoryactions. We aimed to evaluate the safety and efficacy of azithromycin in patients admitted to hospital with COVID-19.Methods In this randomised, controlled, open-label, adaptive platform trial (Randomised Evaluation of COVID-19Therapy [RECOVERY]), several possible treatments were compared with usual care in patients admitted to hospitalwith COVID-19 in the UK. The trial is underway at 176 hospitals in the UK. Eligible and consenting patients wererandomly allocated to either usual standard of care alone or usual standard of care plus azithromycin 500 mg once perday by mouth or intravenously for 10 days or until discharge (or allocation to one of the other RECOVERY treatmentgroups). Patients were assigned via web-based simple (unstratified) randomisation with allocation concealment andwere twice as likely to be randomly assigned to usual care than to any of the active treatment groups. Participants andlocal study staff were not masked to the allocated treatment, but all others involved in the trial were masked to theoutcome data during the trial. The primary outcome was 28-day all-cause mortality, assessed in the intention-to-treatpopulation. The trial is registered with ISRCTN, 50189673, and ClinicalTrials.gov, NCT04381936.Findings Between April 7 and Nov 27, 2020, of 16 442 patients enrolled in the RECOVERY trial, 9433 (57%) wereeligible and 7763 were included in the assessment of azithromycin. The mean age of these study participants was65·3 years (SD 15·7) and approximately a third were women (2944 [38%] of 7763). 2582 patients were randomlyallocated to receive azithromycin and 5181 patients were randomly allocated to usual care alone. Overall,561 (22%) patients allocated to azithromycin and 1162 (22%) patients allocated to usual care died within 28 days(rate ratio 0·97, 95% CI 0·87–1·07; p=0·50). No significant difference was seen in duration of hospital stay (median10 days [IQR 5 to >28] vs 11 days [5 to >28]) or the proportion of patients discharged from hospital alive within 28 days(rate ratio 1·04, 95% CI 0·98–1·10; p=0·19). Among those not on invasive mechanical ventilation at baseline, nosignificant difference was seen in the proportion meeting the composite endpoint of invasive mechanical ventilationor death (risk ratio 0·95, 95% CI 0·87–1·03; p=0·24).Interpretation In patients admitted to hospital with COVID-19, azithromycin did not improve survival or otherprespecified clinical outcomes. Azithromycin use in patients admitted to hospital with COVID-19 should be restrictedto patients in whom there is a clear antimicrobial indication